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Abstract 

This paper proposes a novel technique to prove a one-shot version of achievability resuUs in network information 
theory. The technique is not based on covering and packing lemmas. In this technique, we use an stochastic encoder 
and decoder with a particular structure for coding that resembles both the ML and the joint-typicality coders. Although 
stochastic encoders and decoders do not usually enhance the capacity region, their use simplifies the analysis. The 
Jensen inequality lies at the heart of error analysis, which enables us to deal with the expectation of many terms 
coming from stochastic encoders and decoders at once. The technique is illustrated via several examples: point- 
to-point channel coding, Gelfand-Pinsker, Broadcast channel (Marton), Berger-Tung, Heegard-Berger/Kaspi, Multiple 
description coding and Joint source-channel coding over a MAC. Most of our one-shot results are new. The asymptotic 
forms of these expressions is the same as that of classical results. Our one-shot bounds in conjunction with multi- 
dimensional Berry-Essen CLT imply new results in the finite blocklength regime. In particular applying the one-shot 
result for the memoryless broadcast channel in the asymptotic case, we get the entire region of Marton's inner bound 
without any need for time-sharing. 

I. Introduction 

Information theory aims to find optimal reliable communication rates in networks. The combinatorial structure 
of networks makes the problem difficult in general. However one can employ law of large numbers by looking at 
asymptotic behavior of networks for large blocklengths. But this comes at the cost of a long delay. This motivates 
looking at the problem in the so called "finite blocklength regime." The blocklength in this regime is not infinitely 
long, but is sufficiently large for certain CLTs to hold. Originally studied by Strassen lH], there has been a recent 
surge of works on this topic following the results of Polyanskiy et al ^ (see for instance |l3l-|l6l). 

This work was supported by Iran-NSF under grant No. 88114.46. 



Marcti 5, 2013 



DRAFT 



2 



In this paper we consider one-shot network information theory where a single use of the network is allowed. In 
this case the probability of error cannot necessarily be driven to zero. Further, well-known techniques such as joint 
typicality and time sharing are not applicable here. Given an admissible probability of error, our goal is to find 
a characterization of a set of achievable rates that resembles the form of the asymptotic results. There has been 
some previous work along this direction. Wang and Renner Q derive one-shot upper and lower bounds for the 
problem of transmission of classical information over a classic-quantum channel (see also |8|). Recently Verdu has 
proposed a one-shot version of the covering and packing lemmas, and has applied it to a set of classical problems 
in information theory ||9]- 

Our main contribution is a proof technique for deriving the results on the one-shot region. The technique uses 
elementary tools and is not based on extensions of packing or covering lemmas. It is based on a particular 
construction for encoder and decoders that is not ML, but resembles both the ML and jointly typical coders. 
Our proposed decoders are stochastic and intuitively attempt to pass the received symbol through a certain inverse 
conditional distribution. The Jensen's inequality is central to the analysis of the error The technique can be widely 
applied to problems of network information theory. To illustrate this, we derive new results for the problems of 
Gelfand-Pinsker, broadcast channel, joint-source channel coding over MAC, Berger-Tung, Heegard-Berger/Kaspi 
and multiple description coding. The asymptotic forms of these expressions is the same as that of classical results. 
Our one-shot bounds also imply new results in the finite blocklength regime. 

The most related previous work is that of Verdu ||9]. Whereas ||9l proposes a one-shot covering and packing lemmas 
to solve network problems, we propose a direct analysis comprising of a chain of inequalities. By bypassing the 
need for covering and packing lemmas, we can provide bounds for problems that were originally solved using 
mutual covering and packing lemmas in the asymptotic regime. This is helpful because no one-shot extension of 
the mutual covering and packing lemma exists. We discuss this point in more details in Remark |3] 

This paper is organized as follows: In Section we provide some definitions. This is followed by three sections 
that provide application of the technique to different scenarios. In Section |lll] we consider three problems of multi- 
terminal channel coding, namely, point-to-point channel, Gelfand-Pinsker and broadcast channel (Marton). In Section 
ITVl we consider three problems of lossy multi-terminal source coding, i.e. Berger-Tung, Heegard-Berger/Kaspi and 
Multiple description coding. Lastly in Section FVl we study a joint source-channel coding problem of transmission 
correlated sources over a MAC. In each of these problems we provide a one-shot achievability result. Corresponding 
finite blocklength results could be derived from these results. To illustrate this, we have derived such bounds for 
the Gelfand-Pinsker and broadcast channel problems. 

II. Definitions 

Definition 1. Given a pmf px.Y, the conditional information of x given y is defined by 

hp{x\y) = log TTT- 
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Definition 2. For a pinf px,Y,z, the conditional information density i{x] y\z) is defined by 

ip{x; y\z) := log — — — — , 
p(x\z)p{y\z) 

and for general r.v.'s it is defined by 

ip{x; y\z) := log — -{x, y, z). 

a{px\z X- Py\z) 

Whenever the underlying distribution is clear from the context, we drop the subscript p from ip{x]y\z). 

Definition 3. Let li. be a multi-dimensional normal variable with zero mean and covariance matrix V. The 
complementary multivariate Gaussian cumulative distribution region associated with V is defined by 

Q-i(V, e) {x : P(X < x) > 1 - e}. 

We use M and J to denote size of alphabets of random variables M and J, respectively, i.e. M — \M.\ and 
J = \J\. All the logarithms are in base two throughout this paper. 

III. Multi-terminal channel coding problems 

To illustrate the application of our technique to multi- terminal channel coding problems, we study the problems 
of point-to-point channel, Gelfand-Pinsker and Broadcast channels (Marton) in this section. 

A. Point-to-point channel 

We begin our illustration of the one-shot achievability proof with the classical point-to-point channel. Consider a 
channel with the law qY\x ™d an input distribution qx- Let C = {X(1), - - ,X(M)}bea random codebook where 
the elements X{i) are drawn independently from qx (each codeword X{i) is only a single rv). As usual, X{m) is the 
codeword used for transmission of the message m. For the decoding we use an stochastic variation of MAP decoding. 
Instead of declaring the message m with maximal posterior probability as in MAP, the decoder randomly draws a 
message m from the conditional pmf Pm\y^ where P is the induced pmf by the code, PM.Yi^, y) — y[q{y\X{m))^ 
More specifically, 

q{y\X{m)) _ 2'''(s''^(™)) 

The mutual information term iq{y\X{rh)) is computed using pmf qxqY\x that has nothing to do with the pmf 
induced by the code. However the sequence X(rn) itself is random, hence we have used ^'M|y("^|y) (capital P) 
to denote the pmf. 

We refer this decoder as stochastic likelihood coder (SLC), or as stochastic mutual information coder (SMC)0 
The second equality shows that the probability of selecting a message is proportional to two to the power of its 

'The pmf is random due to the random codebook. 

-The reason for introducing two names for apparently the same object will become clear later These decoders will not be the same in other 
problems. 
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mutual information with the received output. So codewords with higher mutual information have a higher chance of 
being selected as the output of the decoder This resembles the widely used joint typicality decoder in the asymptotic 
regime. 

Theorem 1. The expected value of the probability of correct decoding of SLC (or SMC) for a randomly generated 
codebook of size M is bounded from below by 



EcP[C]>E„^,, \ j^-j-. (2) 

Proof: Observe that the joint distribution of random variables factors as, 

and the probability of correct decoding can be written as P[C] = Tim y ^mym^^^^ ™)' hence we have: 

—q(y\X(m))— , (3) 



(4) 



m.y 

= E> q{y\X{\))— r-F7-TT 

2^q{y■,X{l)) 

= ^E^(i)Ec|x(i)g(y|X(l)) (5) 

2tq{y;X{l)) 

> ^Ex(i)q{y\X{l))— ^ 0^q(y■■X(rn)) 



(6) 



2tq{y;X{l)) 

V ^ ' 

E2h{v\^) 
q(x)q(y\x) — , — ^ (8) 

x,y ' ' 

where (|5]l follows from the rule of iterated expectation, (|6} follows from the Jensen inequality for the convex 
function j(x) = — on M+, and Q follows from the following equation for to 7^ 1, 

X 

X X 

where we use the fact that X(m) is independent of XiX) for to 7^ 1 and is drawn from qx- ■ 
B. Gelfand-Pinsker 

Consider the problem of transmitting a message over a state-dependent channel with state information available 
at the encoder, depicted in Fig. [T| Let qs and qY\x,s be the state's pmf and the channel transition probability, 
respectively. 
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Fig. 1. Coding over a state-dependent cliannel. 



1 ) One-shot achievability: 

Definition 4. An M-code for state-dependent channel consists of a (possibly stochastic) encodenp : [1 : M]x5 i— > X, 
and a (possibly stochastic) decoder ip :y ^-^ [\ : M]. 

Tlieorem 2. Given any qu\s '^^^ function x{u, s), there is an M-code for a single use of the channel whose 
probability of correct decoding is bounded from below by 

^''''^ (1 + J-i2*.(^^;S))(i + M^2-^^(u■'Y)) ' ^^^^ 
where J > is an arbitrary integer and q{u, s, x, y) — q{s)q{u\s)\.[x = x{u, s)\q[y\x^ s). Moreover, loosening this 
bound gives the following upper bound on the error probability of the code, 

P[logJ-z,(C/;5) <7 , or 

Zq(C/;r)-logMJ<7]+3x2-T, (11) 

where 7 is any positive number 

Remark 1. // we apply the above result to n copies of a memoryless state dependent channel, we recover the 
asymptotic Gelfand-Pinsker result. In this derivation the first term in the denominator of dlOl l, 1 + J^i2'*^^''^) 
corresponds to a covering lemma in the asymptotic case, while the second term 1 + MJ2^*^'^'^) corresponds to a 
packing lemma. Observe that the first term is proportional to whereas the second term is proportional to MJ. 
Thus the above formula combines covering and packing lemmas at once. 

Remark 2. If we further loosen the first term of eq. (II lb using the union bound, we get Verdu 's bound on this 
problem ^ except for the term 3 x 2"^. This residual term is not of significance in the finite blocklength n-letter 
regime where we choose 7 of the order of\og{n) (see Theorem\3^; the main contribution comes from the probability 
terms. In a concurrent work UOV , Watanabe et.ai, prove an expression similar to eq. (II 11 1 using a different method 
based on channel simulation. They also applied their approach to the problem of source coding with a helper and 
to the Wyner-Ziv problem. It is not clear whether their approach is applicable to the scenarios such as broadcast 
channel, multiple description coding, etc that is solved in the asymptotic case using the multivariate covering 
lemma, since no extension of channel simulation (based on the work of Cuff KTTH j is known for multiuser scenarios. 
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Nonetheless, our technique bypasses the need for either an extension of covering lemma to multivariate covering, 
or a multi-terminal extension of the channel simulation result. See also Remark |5] 

Proof: 

Let C = j^i be a random codebook whose elements are drawn independently from qjj. Here J 

is introducing redundancy but since it will be decoded at the receiver we can view it as a dummy message. 

Encoding: Instead of using conventional random covering, we use an SMC which acts as follows. Given m and 
s, the SMC chooses an index j with the probability 

2^q(s■,U[m,j)) 

Then the encoder transmits x{U{m, j), s) through the channel. Observe that the above SMC resembles a joint- 
typical encoder of the asymptotic regime. Given m and s, the higher the information between U{m,j) and s, the 
more likely we choose it at the encoder 

Decoding: In contrast to the point-to-point problem, computing the error probability of SLC is challenging. An 
SLC uses the induced Pm.j\y by the code. Instead, we use an SMC with the following rule for decoding. Observing 
y, decoder uses the following SMC to find both the message m and the dummy message j: 

2'-<iiV\U{m,~j)) 

PDec(m,j|y) = ^ _o^,(.s-Mm3)) - 

Analysis: We declare an error if (m, j) ^ Observe that the joint distribution of random variables factors 

as, 

^AUSYMji^^j^ ^' '^'j) = -^<l{s)PEucij\m, s)q{y\U{m, j), s)PDecim, j\y), 

where q{y\U{m,j),s) = qY\x,s{y\x{U{m,j),s),s). The probability of correct decoding is P[C] = 
PMjSYMji™d,s,y,m,j); hence we have: 

TTlis) -, -^q(y\U(m,j),s)— , (12) 

= E> Jg(s) -^q(y\U(l,l),s)— , (13) 

s,y /-^j ^ ^m,j 

_ / Jofs)2'(''''^(i'i)) 2*(2'''^(i'i)) A 

^ l «c.u',Ei^-'--'-^^'- ''^'^''-''"' En.....,E.,.,2...^.'....... j 



> 



l^^uii,!) (^ 2H^;^(M))+j g(^^'^(^'^)'^) 2»fea(i.i)) + Mj j ^^^^ 



J 2*(2''") 

2^ q{u, s, y)2,(,,„) _^ J • 2»{y;n) + mJ ^^^"^ 

1 

= ^USY ^ j-i2^(a;S))(i + MJ2-*(t^^^)) ' ^^^^ 
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where (fTSl l is due to symmetry, the main step ( fT4b follows from Jensen inequality for the two-valued convex function 

f{xi,X2) = on R^, ( fTSl ) follows from the fact that U{i,j) is independent of C/(l, 1) for 7^ (1, 1) and 

X1X2 

generated according to qu, and ( fT6b follows from the fact that [7(1, 1) is generated according to qjj- 
Deriving the loosened bound (IIH ; 

1 



{1 + }-^2<u.s))(^\ + U}2-<u,Y)) 



1 



1 {log J - i{U; S) > 7, i{U; Y) - log M J > 7} 



> 



1 



(1 + 2-7): 



(l + J-i2'(t^^^^))(l + MJ2-«(c^;^)) 
rP[logJ-i(C/;5) >7, ^(C/; F) - log MJ > 7] (19) 



P[E]<1- j^^^-^P [logs -i{U-S)>^, <f/;y)-logMJ>7] 

= P [log J - i{U; S) < 7, or ^(C/; Y) - log M J < 7] 

1 



(20) 



1 



(l + 2-'/)2 

< P [log J - liU; S) < 7, or z{U; Y) - log MJ < 7] + (^1 - j^^^^^ 

< P [log J - i{U; S) < 7, or ^(C/; Y) - log M J < 7] + 3 x 2"''. 



P [log J - i{U; S) > 7, i{U; Y) - log M J > 7] (21) 

(22) 



(23) 



2) Second Order achievability of Gelfand-Pinsker channel: 

Theorem 3. Given a memoryless state-dependent channel (qs, qY\x.s) with state known non-causally at the encoder, 
for any ((7(715, a;(M, s)), the following rate is [n^e)- achievable: 



where 



and 



R = /([/; Y) - I{U; S) - -^Rd - O 

Vn \ n 



Rd = min R, 

R:3R, s.t. [fl,/?-fl]Tee-i(VGP,e) 



(24) 



(25) 



Vgp = Cov 



^{U;S) 
i(U;Y) 



(26) 



Proof: We apply ( fTTl i to n use of the channel. Assume that qjjn[u'^) = n"=i 9c/ (""«)' ^o (C/",S'",F") are 
i.i.d.. Substituting l^S"^ ( fTTI ) implies: 



P[f] < P ( logJ -^(C/";S'") < ilogn, or z([/"; F") - log MJ < ^ logn 

V 2 2 y V'^ 

Given e > 0, finding a code such that for some e' < e 



(27) 



e' = P ( log J -?([/"; S*") < i log n, or i(t/"; F") - log MJ < i log n ) + 4=- (28) 

2 2 1 Jn 
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Fig. 2. Coding over a broadcast channel. 



implies that e is an upper bound on P[£]. Equivalently, we would like to find a code such that 

l-e'-A = p(^logj_z({/";5")>ilogn, z(C/"; F") - log MJ > 1 logn^ . (29) 

Let log J = nI{U; S) + y/nR + ^ log n and log M = n{I{U; Y) - I{U; S)) - ^/nR - log n. The random variables 



i(C/"; S*") and y") are sum of i.i.d. random variabels. Applying multi-dimensional Berry-Essen CLT 11201 to 
implies the following equivalent form: 



1 



Gi 

G2 



< 



R 

R-R 



(30) 



where G = [Gi G2Y is a multidimensional normal rv. with zero mean and CovG — Vgp- Using the definition of 
Q^^(V, e) and smoothness of distribution of normal rv., we get 

/ log n ^ 



[R,R~rY e Q-\YQP,e') + 



(31) 



Taking the limit of e' ^ we see that for any arbitrary [R,R — R\^ in Q^^(VGP;e), we can achieve P[£] < e. 
This completes the proof. ■ 



C. Broadcast channel 

Consider the problem of transmission of private messages over a broadcast channel depicted in Fig.|2] Let QyiY^Ix 
be the channel transition probability. 
1 ) One-shot achievability: 

Definition 5. An (Mo, Mi, M 2) -cot/e /or broadcast channel consists of a (possibly stochastic) encoder ip : [1 : 
Mo] X [1 ; Ml] X [1 ; M2] 1-^ X, and two (possibly stochastic) decoders V'fc : 3^fc ^-^ [1 : Mo] x [1 : Mfc], fc — 1, 2. 

Tlieorem 4. Given any qua,Ui,U2 and function x(uo, ui, U2), there is an (Mo, Mi, M2)-code for a single use of the 
channel whose probability of correct decoding is bounded from below by 



E 



1 + (JiJ2)"^2'(^i'^^l^")) + MfcJfc2-*(^'''^'='^'') + MoJfcMfc2-'(^o^'''^'=)) 



fc=i 
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where Ji, J2 > are arbitrary integers. Moreover, loosening this bound gives the following upper bound on error 
probability of the code, 

P[logJiJ2-z([/i;C/2|f/o) <7 , or 

j(C/i;Yi|i7o)-logMiJi < 7 ,or 2(f/oC/i; Yi) - log MqMiJi < 7 or 

?(C/2;Y'2|C/o)-logM2J2 <7 ,or ^(C/oC/2; Y'2) - log M0M2J2 < 7] + 17 x 2"'^, (32) 
where 7 is any positive number 

Corollary 1. Given any qui.U2 and function x{ui,U2), there is an (Mi, M2)-code for a single use of the channel 

whose probability of correct decoding is bounded from below by 

2 



E 



1 + (JiJ2)"'2*(^i'^^n 11(1 + MfcJfc2-*(^^'^'')) 



fe=i 

where Ji, J2 > are arbitrary integers. Moreover, loosening this bound gives the following upper bound on error 



probability of the code. 



P[logJiJ2 -«(C/i;?72) < 7 , or 
i(C/i;Yi)-logMiJi <7 , or 

«(C/2;>"2)-logM2J2 <7] +7x 2-^ (33) 



where 7 is any positive number 



Remark 3. Verdu derives a one-shot bound for the same problem in [9^. He derives the bound by proposing 
a one-shot covering and packing lemma. However to get access to the boundary of Marton 's inner bound one 
needs a mutual covering lemma ( since time sharing is not possible in one-shot and not useful in finite block length 
regime). For this reason Verdu' s result seems to be weaker than ours. Our technique allows us to bypass the need 
for developing a one-shot version of the mutual covering lemma. 

Proof: For simplicity we prove the one-shot version of Marton with two auxiliaries (we will put the full proof 
in the next version of this draft). We only show the lower bound on probability of correct decoding. Derivation of 
the loosened bound is similar to that of Gelfand-Pinsker and thus omitted. 

Random codebook generation: Let C = Ci XC2 = {C^i(wi, Ji)}mi'ii ^ {U2{'>n2, j2)}^.^=i^j^=i be a random 
product codebook, in which the codebooks Ci and C2 are generated independently and the elements of the codebook 
Cfcjfc — 1,2 are drawn independently from qu^. Thus the codebook is generated according to ru^u^ — quiQu-A 
qjji U2 ■ Here Ji , J2 are introducing redundancy but since it will be decoded at the receiver we can view these as 
dummy messages. 

Encoding: Instead of using conventional mutual covering, we use an SMC which acts as follows. Given mi, 777-2, 
the SMC chooses indices ji,j2 with the probability 

2«9(fi(™i,ii);t'2(m2 J2)) 

-PEnc(il,i2|™l,»772) = — ^-77- T"-- (34) 

^jl ,32 
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Then the encoder transmits x{Ui{'mi, ji),U2{m2, j2)) through the channel. Observe that we generate codewords 
according to riiiU2 but compute the informations ig using quiU2- This resembles the Marton coding scheme where 
we generate and C/^ independently but choose the jointly typical ones for transmission. 

Decoding: We again use an SMC for decoding. Observing yk, decoder k uses the following SMC to find both 
the message and the dummy message jt: 

Analysis: Observe that the joint distribution of random variables factors as, 

-P(to1:2, jl:2,yi:2,TOl:2, jl:2) = fEnc(jl:2 |to1:2) 

2 

q{yi:2\Ui{mi,ji),U2(m2,j2)) Y[ PDeck{mk,jk\yk), 

k=l 

where q{yi:2\Ui{mi,ji), 1/2(1712, j2)) is equal to qYi.2\x{yi:2\x{Ui{mi,ji),U2{m2,j2)))- We make an error 
if either of the decoders fail. The probability of correct decoding can be bounded from below by P[C] > 
Smi:2ji:2,yi ^("^i-2'Ji:2,yi,y2, Mi:2 mi:2,Ji:2 = ji:2) , hence skipping similar symmetry ai-guments we 
have: 

EP[C] > E V J1J2 , , ~ . — — g(2/i|?7i(l,l),f/2(l,l)) TT ^-FT-, ^ (35) 

yi,y2 ^jlj2 k=l ^rnk,3k 

( j^j22*5('^i(i.i);'^2(i,i)) 



Vl,y2 \—l-\Ul:2\±;l-) J 



""'^^""'"'n .,,..,.,,s,.,........<»...» j ™ 

^ / j^j22»,(c/i(i.i);t/2(i4)) 2 2'" (2"=;'^'= \ 

> L I 2.,(c/.(i.i);t/2(i,i)) + j^j/(2^i|f^i:2(l, 1)) 11 2.,te.;t/.(i,i)) + MfeJfe J ^^^^ 



ai,a2 \ /c=i 

_ J1J2 -A- 2'"^"'"'°) 

- 2^ ^*'"l''"2'yiJ2J,(ui;u2) ^ J J 11 2»,fefc;«fc) + Mfejfe 

f l,"2,J/l,J/2 fc=l 

~ ^9 7~~7TT3r:^7777r7nTTr2 



(38) 



(39) 



(1 + (JiJ2)-i2».(^^i;t^^)) nLi(l + MfeJfe2-».(c^'=;^'=)) 
where (|35] l is due to symmetry, the main step ( [36l l follows from Jensen inequahty for the three-valued convex 

function f{xi,X2,X3) — on M"^. The expectation in (|36] | is over Ui-9{1,1) of the codebook generation 

X1X2X3 

distributed according to rjj^ij^. Equation ( l37T i follows from the following equations: 



"2 

= ^q(w2|f/(l,l)) = l, 32^1, (40) 

"2 

Ec|c/,.(i,i)2'''(^^(^^^'^)^^^(i'i» = 1, .71 ^ 1, (41) 
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Ec|c/,.(i,i)2'''(^^(^'^'^)^^^(^'^'^» = 1, ji ^lj2^ 1, (42) 

where in ( |40| | we use the f act that [/2 ( 1 , j2 ) is independent of([/i(l,l),C/2(l,l)) for ^2 7^ 1 and generated according 
to qu2- (ED and ( l42b follows similarly. Finally and ( |38] l follows from the fact that 1), ?72(1, 1)) is generated 

according to product distribution rjj-^u^ = qui<lU2- ■ 

Remark 4. Unlike the Gelfand-Pinsker problem, the SMC encoder used is ( I43l l cannot be written in the form of a 
SLC encoder The SLC encoder has the following form: 



jSLC 



(jl,j2|TOi,m2) 



q(Ui{mi, ji),U2{m2, j2)) 
q{Ui{mi,ji),U2{m2,j2)) 



We use SMC instead of SLC, because the analysis of SLC is challenging. 

2) Second Order achievability of broadcast channel: 

Theorem 5. Given a memoryless broadcast channel qy^.Y^lx, for any ((7;7^[/2 1 2;(wi, M2)), the pair _/?2) i^ 
{n,e)- achievable, if there exists (_Ri,i?2) such that 

Ri + i?2 



— i?i — Ri 
— i?2 ^ R2 



Jn \ n 



where 



in which 



Ibc = '^UxU-iYxYi^^C; Vbc = Cov(/bc), 



'BC 



^{Ul■.U2) 
-i{U2;Y2) 

Proof: The proof uses (|33] | in a way similar to the proof of Theorem |3] and hence is omitted. 



(43) 



(44) 



(45) 



IV. Multi-terminal lossy source coding problems 

To illustrate the application of our technique to multi-terminal lossy source coding problems, we study the 
problems of Berger-Tung, Heegard-Berger/Kaspi and Multiple-description in this section. Throughout this section 
we use s for source and s for its reconstruction. Since s is created from a set of rv's available at the decoder, 
we follow El Gamal and Kim's notation fT2l to also use s as a function of the rv's available at the decoder For 
instance if the decoder has rv Y , we use a decoding function s{y). 
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Fig. 3. Distributed lossy compression system 



A. Berger-Tung 

Consider the problem of distributed lossy compression depicted in Fig. [3] Let ^e the joint distribution of 

the sources and dk{skTSk),k — 1,2, be two distortion measures. We will use the probability of excess distortion 
as the criterion for measuring the reliability of the system. 

Definition 6. An (Mi, M2)-code for distributed lossy compression consists of (possibly stochastic) encoders ipk '■ 
Sk > [1 : Mfe],fc ~ 1,2, and (possibly stochastic) decoder : [1 : Mi] x [1 : M2] 1— > 6*1 x 82- Given a pair of 
distortion levels (Di, D2), the probability of excess distortion is defined by, 

P[f;Di,D2] := P[di(^i,§i) > Di V d2(52, ^2) > D2]. 

Also, we define the probability of no-excess distortion by 

P[C;Di,D2] := P[di(^i,5i) < Di, 62(82, S2) < D2]. 

We prove a one-shot version of the result of Berger-Tung llTSI , lfT4ll for this problem. 

Tlieorem 6. Given any pmf qij-^\Si1U2\S2 and functions Sfe(wi, U2), fc — 1, 2, there is an (Mi, M2)-code for a single 
use of the sources whose probability of no-excess distortion is bounded from below by 

l{di(5l,Sl([/l,f/2)) < Di,d2(52,S2(C/l,t/2)) < D2} 



P[C;Di,D2] >]E 



(1 + i2»(Si;C^i))(l + J-i2^(S2;C/2)) (1 + (j^M^i + JiM^^ + Jih{MiM2)-^)2-<Ui;U2)\ ' 

(46) 



where > Mfc,fc = 1,2 are arbitrary integers. Moreover, loosening this bound gives the following upper bound 
on the probability of excess distortion of the code. 



P[di(5i,Si(C/i,t/2)) > Di,Ord2(52,S2(t/i,{/2)) > D2,or 

15 X 2^^. (47) 



J1J2 

log Ji - 1(5*1; t/i) < 7,or log J2 - j(5'2; J72) < 7:Or i{Ui; U2) - log . . . . < 7 
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Remark 5. The term 1 + Jj, ^2'*^'^'='^'=^ fc = 1, 2 corresponds to a covering of Sk with Uk in the asymptotic regime. 
The alphabet size ofUk is Jfc. We use a random binning ofUu, mapping it from a set of size Jfe to a set of size Mfe. 
This explains the inequality ik > Mk for k = 1,2. The term (l + (J2M2'^ + JiMj^^ + JiJ2(MiM2)"^)2-*(^i'^2)) 
corresponds to a mutual packing lemma in the asymptotic regime. 

Proof: We only prove the lower bound on probability of no-excess distortion. Derivation of the loosened bound 
is similar to that of Gelfand-Pinsker and is thus omitted. 

Random codebook generation: Let C = Ci x C2 = {C^i(ii)}j^=i x {U2{j2)Yj\^i be a random product codebook, 
in which the codebooks Ci and C2 are generated independently and the elements of the codebook Ck, k — 1,2 are 
drawn independently from qu^ . Thus the codebook is generated according to rjji U2 — QUi QU2 QUi U2 ■ 

Random binning: Let Bk ■ [1 ■ Jfc] >-> [1 : Mk],k = 1,2 be two independent random mappings (binning), in 
which Bk maps each element of [1 : J^] uniformly and independently to the set [1 : M^:]. 

Encoding: Encoder k ^ 1,2 uses an SMC followed by a random binning to obtain the index mk- First given 
Sk, the SMC chooses an index jk with the probability 

2*<j(sfc;t^0fc)) 

Then the encoder transmits nik = Bk(jk) to the decoder 

Decoding: We use a modified SMC for decoding w.rt. a receiving indices (7711,7712). Observing (7^1,7712), 
decoder uses the following modified SMC to find both the ji and j2 and thus {Ui, U2): 

^ ,, , , , T^('^^O^y'U-02))i{Biiji)^rni,B202)=rri2} 

Po..{n,n\rm,m2) - ^.,iu.uM)H{B,(j,) = m„ B2{h) = -2} ' 

Remark: Observe that the above SMC resembles a joint-typical decoder for the mutual packing lemma in the 
asymptotic regime. It can be considered as a dual of the SMC encoder of equation (l43l l that corresponded to a 
mutual covering lemma in the asymptotic regime. 

Given 7711 and 7712, the decoder chooses among the pairs {Ui{ji), 1/2^2)) assigned to the bin {1711,1712). The 
higher the information between Ui{ji) and U2{j2), the more likely we choose it at the decoder Finally, decoder 
computes Sk{Ui{ji), 1/2^2)) as the estimate of Sk- 

Analysis: Observe that the joint distribution of random variables factors as, 

2 

P{si:2,mi;2,jl:2,jl:2) = ^C^l , 52 ) J]^ ^EnCfc (jfe | Sfc) (ji ) ^ nil, B2{j2) = ™2}f'Dec O'l , ^2 |™1 , »7l2 ) . 

fe=l 

We consider two error events: 

1) The decoder fails to recover the correct pair (71,^2), i-e. (^1,^2) 7^ {ji,j2). 

2) One of the distortions corresponding to the pair (ii,j2) exceeds the designated levels, i.e. 
dik{sk,Sk{Ui{ji),U2U2))) > Dfe for some k e {1,2}. 

The probability of correct decoding can be bounded from below by 



P[C] > ^ P{si:2.mi.,2,jl:2,Jl=jl,J2^j2) 



S 1:2, m 1:2 J 1:2 
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X l{di(si,Si([/i(ji),t/2(j2))) < Di,d2 (s2,S2(C/l(jl),(72(j2))) < D2} , 

hence skipping similar symmetry arguments we have: 

2i{si;Ui{l)) 2'(''2;C/2(1)) 

EP\C]>EcB V MiM2JiJ2g(si,S2) -, l(Bi(l) = 1) ^1(^2(1) = 1) 

Sl,S2 ^jl ^32 

Ej,j2 2<v.(hyM))i{B^Q^) = 1,B2(J2) = 1} 

X l{di(si,Si(f/i(l),t/2(l))) < Di,d2(s2,S2(C/l(l),C/2(l))) < D2} (48) 



> M1M2 ^ g(si,S2)E;7i.2(l),Bi 



2(1) 



2.(c/i(i);c/2(i))i(^^(l) = i,B2(i) = 1) 



> 



X l{di(si,Si([/i(l),[/2(l))) < Di,d2(s2,S2([/l(l),C/2(l))) < D2}^ (49) 

2i(si;(7i(l)) 2'(''2;(/2(l)) 



I1M2 ^ (7(si,S2)E,7i^2(l),Bi,: 



(1) 



1 + j-i2*("i't^i(i)) 1 + j-i2<''2;'^2(i)) 

2«(c/i(i);t^2(i))i(5^(l) = 1,^2(1) = 1) 



2<Vi{iy,U2{i))^Bi{l) = 1,B2(1) = 1) + J2M^il(Bi(l) = 1) + JiMf 4(^2(1) = 1) + JiJ2(MiM2)-i 

X l{di(si,Si(;7l(l),{/2(l))) < Di,d2(s2,S2(t/l(l),C/2(l))) < D2} ^ (50) 

= q{si,S2)q{ui\si)q{u2\s2) , } , r , } , r 

Si,S2,Ui,Jl2 J- 2 

Ijdi (■si,si(mi,M2)) < Di,d2 (52,52(^1,^2)) < D2} 
1 + (J2M^i + JiMr' + JiJ2(MiM2)-i)2-»(«i;«2) 



^ l{di(gi,Si([/i,[/2)) < Dl,d2 {S2,S2{UuU2)) < D2} 

9si,2^'i:2 _^ j-i2'('5i;c^i))(l + J-i2<S2;C/2)) (1 + + JiMj^i + JiJ2(MiM2)-i)2~Hc^i;t/2)) 

where (l48T l is due to symmetry, the main step ( |49] l follows from Jensen inequality for the three-valued convex 

function f(xi,X2,X3) — ^ on Mi. The expectation in ( |49] l is over C/i.2(l,l) of the codebook generation 

X1X2X3 



distributed according to ru^u^. Equation ( l50b follows from separating the cases [ji — l,j2 7^ 1), (ji 7^ 1,^2 = 1), 
(ji 7^ 1,J2 7^ 1) and (ji = 1,^2 — 1)- This is discussed below. Lastly, equation ( |52] | follows from the facts 
that random codebook and random binning are independent and (Ui{l), 1/2(1)) is generated according to product 
distribution rjy^jy^ = quiqu2- 

Derivation of equation dSOb ." In the following we consider each of the four case mentioned above separately. 

. Case 1, (Ji = 1,32^ 1): 

Ec.6|c/.2(i)A:2(i)2^^''^^'^^''=^'^^^^l{5i(l) - hB2Cj2) = l} = 5]g(ii2)2^(^^(i)^"^)l{Si(l) - 1}M^' 

«2 

= M2^i^g(u2|C/i(l))l{i?i(l) = 1} = M^'1{B,{1) = 1} (53) 

"2 

where in ( |53] l we use the following facts: 
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Fig. 4. Lossy source coding when side infonnation is absent. 



1) ^^2(^2) is independent of ([/i(l), L/2(l)) for j2 7^ 1 and generated according to qjj^. 

2) The random binning and random codebook are independent and 52(^2) has a uniform distribution over 
[1 : M2]. 

• Case 2, [ji 7^ 1,^2 = 1): This is similar to the previous case and gives us 

Ec.6|c/..(i),B,.(i)2'(^^(^^^)^^^(^»l{Bi(ji) - 1,52(1) = 1} = Mril{B2(l) = 1} (54) 

• Case 3: (ji 7^ 1,^2 7^ 1): In this case Ui{ji), 1/2^2) are independent of ?7i:2(l), i3i:2(l), and are generated 
according to qu^qu^. 

Ec,6|[/..(i).i3..(i)2^^^^(^^^)^^^(^^^»l{Si(ji) = 1,S2(J2) = 1} = E 9("i)'Z(^2)2'("^^"^)(MiM2)^i 

Ui ,U2 

= (MiM2)-^ E = (MiM2)-^ (55) 

Ml ,U2 

, Case 4: (ji = l,j2 = 1): 

IEc.eit/i..(i)A.(i)2'(^^('^^^^^'»l{Bi(l) = l,i?2(l) = 1} - 2^(^^(i)^^^(i»l{Si(l) = 1,^2(1) = 1}. (56) 

■ 

B. Heegard-Berger/Kaspi 

Consider the problem of lossy source coding when side information may be absent, depicted in Fig. H] Let qs 
be the distribution of the source and dfe(s, Sk), k — 1,2, be two distortion measures. We will use the probability of 
excess distortion as the criterion for measuring the reliability of the system. 

Definition 7. An M-code for distributed lossy compression consists of (possibly stochastic) encoder ip : S t-^ [1 : M], 
and (possibly stochastic) decoders : [1 ; M] i— )• Si and ■(/;2 : [1 : M] x 3^ i— > 52- Given a pair of distortion levels 
(Di, D2), the probability of excess distortion is defined by, 

P[£; Di, D2] := P[di(5, ^1) > Di V d2(5, ^2) > D2]. 

Also, we define the probability of no-excess distortion by 

P[C; Di, D2] := P[di{S, Si) < Di, d2{S, S2) < D2]. 



March 5, 2013 



DRAFT 



16 



Remark 6. Setting Di = oo, this problem reduces to the Wyner-Ziv problem. A one-shot result for the Wyner-Ziv 
problem has been concurrently obtained by [10]. 

We prove a one-shot version of the result of Heegard-Berger/Kaspi ifTSll . fT6l for this problem. 

Theorem 7. Given any qwu\S and functions si{w), §2(10 ^u,y), there is an M-code for a single use of the source 
whose probability of no-excess distortion is bounded from below by 

pr^ n n 1 > ¥ i{di{S,h{W)) < D,,d2 iS,S2iW,U,Y)) < D2} 

where Mi,M2,J2 ore integers such that M = M1M2 and J2 > M2. Moreover, loosening this bound gives the 
following upper bound on the probability of excess distortion of the code, 



P[di(^,si(M^)) > Di,or d2 {S,S2{W,U,Y)) > D2,or 

logMi - i{S;W) < J, or logMih - i{S;WU) < j, or i{Y;U\W) ~ log hM^^ < 1 



5x2-'^. (58) 



Proof: We only prove the lower bound on probability of no-excess distortion. Derivation of the loosened bound 
is similar to that of Gelfand-Pinsker and is thus omitted. 
Random codebook generation: 

* Ci ~ is a random codebook whose elements are drawn independently from qw- 

• For each mi e [1 : Mi], let C2(mi) = {U{mi,j2)}^l^i be a random codebook whose elements are 
drawn independently from qu\w{-\W{iT^i))- Moreover the codebooks C2{mi),nii G [1 : Mi] are generated 
independently. 

« Let C be the set of all codewords. 

Random binning: Let S : [1 : J2] 1-^ [1 : M2] be a random mapping (binning), where B maps each element of 
[1 : J2] uniformly and independently to the set [1 : M2]. 

Encoding: Encoder uses an SMC followed by a random binning to obtain the pair {mi, 1712). First given s, the 
SMC chooses a pair (mi, ^2) with the probability 

2iq(s:W(mi),U(mi,j2)) 
PE.c{mi,j2\s) = — 2,^^s;W{m,),U{rhuh))- 

Then the encoder computes m2 = B{j2) and sends m = {mi,m2) to the decoders. 

Decoding: Decoder 1 outputs si{W{mi)) as the estimate of S. Decoder 2 uses a modified SMC for decoding 
w.rt. a received index (mi,m2). Observing (7711,7712), decoder 2 uses the following modified SMC to find j2 and 

thus (Vl^(777l), C/(777l, ^2)): 

PDMmi,m2) - ^^^^ 2^qiymm.,h)m,n.))^BCj2)=rn2y 
Observe that the above SMC resembles a joint- typical decoder of the asymptotic regime. Finally, decoder 2 computes 
S2{W{mi),U{mi,j2),y) as the estimate of S. 
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Analysis: Observe that the joint distribution of random variables factors as, 

P{s,y,mi:2,j2,j2) = q{s,y)PEnc{mi, j2\s)l{B{j2) = m2}PDec{j2\mi,ni2). 

We consider two error events: 

1) Decoder 2 fails to recover the correct j2, i.e. j2 j2. 

2) The distortion of decoder 1, or that of decoder 2 corresponding to j2 exceeds the designated levels, i.e. 
di(s,si(VF(mi)))) > Di ord2(s,S2(W^(mi),C/(mi,j2),y)) > D2. 

The probability of correct decoding can be bounded from below by 

P[C]> -P(s.y>"^l:2,j2,i2 =J2) 

X l{di(s,si(W^(mi))) < Di,d2(s,S2(M^(mi),C/(mi,j2),y)) < D2} , 
hence skipping similar symmetry arguments we have: 

EP[C]> ^c^.J:M^M2^2,is,y) ^_^ J.)) WD = D 2^i,.ua3.nwm^B(j,) L 1) 

X l{di(s,si(W^(l))) < Di,d2(s,S2(W^(l),[/(l,l),y)) < D2} (59) 
/ M^j22H'';W'(i).^(i'i)) 



> M2^g(s,y)EH'(i),(7(i,i),B(i) 



2»(y;(7(l,l)|W(l))j_(^^2) = 1) 



IEc,B|w(i).c/(ia),B(i) E], 2<y■Wl,h)ml))^Bij2) = 1) 
X l{di(s,si(W^(l))) < Di,d2(s,S2(W^(l),f/(l,l),2/))< D2} ) (60) 



> 



M2 2^g(s,y)Ety(i),c/(i,i),B(i) (^MiJ2 + J22'(^'^(i)) + 2'(-;^(i)'^(i^i)) 



Q{s,y)q{w,u\s) l—jp 

t,y,w,u ^ 1 



2<viU(i.i)\w(i))i(^^^l^ ^ l-^ 
2H?/;C/(i,i)|M/(i))i(5(l) ^ 1) + J2M^i 

X l{di(s,si(VF(l))) < Di,d2(s,S2(VK(l),;7(l,l),y)) < 02}^ (61) 
1 1 



-l2i(s;«') _|_ (^M J2)"l2'(*'"''") ' 1 + J2M^-^2^*(2''"I'") 

X l{di (s,si(u')) < Di,d2(s,S2(w,w,y)) < D2}j (62) 

^ l{di (^,^i(T4^)) < Di,d2(^,J2(W^,t/,y)) < D2} 

(1 + i2*(S;M') + (MiJ2)-i2*(^'^'^)) (1 + J2M^i2-»(^;t^|i^)) 

where ( [59] l is due to symmetry, the main step ( |60] | follows from Jensen inequaUty for the two-valued convex 

function f(xi,X2) — on R^. Equation (l6Tl l follows from computing the denominator terms. This is discussed 

2:1X2 

below. Lastly, equation ( |62| | follows from the facts that random codebook and random binning are independent and 
U{1, 1)) is generated according to qwu- 
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Fig. 5. Multiple description coding. 



Derivation of equation (|6TI ); Deriving this equation involves computing two denominator terms. To compute 

^CMW{i),u(.i,i),B[i) Ej, 2^(«;^(i'^^^)l^(i))l(B(j2) - 1) for any ^ 1 we have: 

U 

= M2-i^g(u|W^(l),y) = M2-i, (64) 

u 

where we have used similar argument used in the proof of (l5Jt . 

To compute '&c\w{i),u{i,i) X^mi 2'^*'^'^™!^''^'^™!'^^^^ we consider the following three cases separately: 
. Case 1, (ttii = 1,^2 7^ 1): 

where we have used the fact that ?7(l,j2),j2 7^ 1 is independent of and drawn from 

qu\w{-\W{l)). 

• Case 2, mi 7^ 1: 

where we have used the fact that U{rhi,j2), mi 7^ 1 is independent of W(l), U{1, 1) and drawn from qwu- 

• Case 3, (mi = 1,^2 — 1): 

^C\W(l),U(l,l)^ w, ^ , ; ^ 2 ^ ' ((,7) 



C. Multiple Description 

Consider the problem of multiple description coding depicted in Fig. |5] Let qs be the distribution of the source 
and dfc(s,Sfc),fc — 0,1,2 be three distortion measures. We will use the probability of excess distortion as the 
criterion for measuring the reliability of the system. 
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Definition 8. An (Mi, M2)-code for the multiple description coding consists of (possibly stochastic) encoder Lp : 
S t-^ [I : Ml] X [1 : M2], and (possibly stochastic) decoders ^0 : [1 : Mi] x [1 : M2] 1-^ Sq, tpi : [1 : Mi] 5*1 and 
■02 : [1 : M2] ^-)■ 82- Given a tuple of distortion levels (Dg, Di, D2), the probability of excess distortion is defined 

by, 

P[£-;Do,Di,D2] := P[do(5,§o) > Do V di(^,^)>Di V d2(5, ^2) > D2]. 
Also, we define the probability of no-excess distortion by 

P[C;Do,Di,D2] := P[do(5,S^o) < Do,di(5,§i) < Di, A2{S,S2) < D2]. 

We prove a one-shot version of an equivalent characterization of Zhang-Berger inner bound llT7l . ifTSi for this 
problem. 

Theorem 8. Given any quaUiU2\S and functions sq{uq,ui,U2), si{uq,ui) and S2{uo,U2), there is an (Mi,M2)- 
code for a single use of the source whose probability of no-excess distortion is bounded from below by 

l{do(5,so((7o,C/i,C/2)) < Do,di(5,si(t/o,C/i)) < Di,d2 {S,s2{Uo,U2)) < D2} 



P[C; Do,Di,D2] >E 



(68) 



where Jo is a common divisor of Mi and M2. Moreover, loosening this bound gives the following upper bound on 
the probability of excess distortion of the code, 

P[do(5,Jo(f/o,t/i,C/2)) > Do,ordi(5,si(C/o,t/i)) > Di, or d2 (5, J2({/o, (72)) > D2,or 

logJo - ^S*; t/o) < 7,or log Mi - «(S'; J/of/i) < 7, or log M2 - «(S'; C/of/2) < 7, or 

+ 4x2"^. (69) 



log MiMi _ ^(5; U0U1U2) - ^(^7l; U2\Uo) < 7 
Jo 



Proof We only prove the lower bound on probability of no-excess distortion. Derivation of the loosened bound 
is similar to that of Gelfand-Pinsker and thus omitted. Since Jo is a common divisor of Mi and M2, there exists 
positive integers Ji and J2 such that Mi = JqJi and M2 = JoJ2- 
Random codebook generation: 

» Cq ~ {Uo{jo)}'j'^^i is ^ random codebook whose elements are drawn independently from qij^. 

. For each jo E [1 : Jo], let Ci2(jo) = Ci(jo) x C2(jo) - {Ui{ji)}jl^, x {U2{j2)}f,^i be a random 
product codebook, in which the codebooks Ci(jo) and C2(jo) are generated independently and the elements 
of the codebook Ck{jo),k — 1,2 are drawn independently from qUk\Uoi-\Uo{jo))- Moreover the codebooks 
Ci2(jo), jo G [1 : Jq] are generated independently. 

« Let C be the set of all codewords. 

Encoding: Encoder uses an SMC for three rv.'s to obtain the tuple (jo, ji, ^2)- Given s, the SMC chooses a 
tuple (jo, ji, ^2) with the probability 

2Hi^\Uo{jo),UiUo,ji),U2{3o,32))+i,,{Ui(3o,n)\U2{3o,j2)\Uo{l)) 
fEnc(jo, jl, j2|s) - 2''i(''''^oGo),U^GoJi),U2{joj2))+t,{Ui{joJi);U26oj2)\U„{]o)) ■ 

Then the encoder sends mi — (jo, ji) and m2 = (jo, j2) to the decoders. 
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Decoding: Observing (mi, 777,2) = {ja, 31,32), decoder computes so(t/o(jo), C^iOo, ji), t^2(jo,i2)) as the 
estimate of S. Observing nrik — {jo,jk), decoder k — 1,2 computes SQ{Uo{jo),Uk{jo, jk)) as the estimate of 
5. 

Analysis: Observe that the joint distribution of random variables factors as, 

P{sJo,jl,j2) = '7(s)PEnc(jo,il,i2|s)- 

The probability of no-excess distortion can be bounded from below by 

P[C] > ^ PisJoJlj2)Xs,UoUo),UiUo,Ji),U2Uo,j2)' 

where 

Xs,«o,"i,u2 ■= 1 {do (5,50(^0,771,772)) > Do,di (5, si(77o,77i)) > Di,d2 {S, 52(770,772)) > D2} . 
Skipping similar symmetry arguments we have: 

2i{s;Uoil),Ui{l.l):U2il:l))+iiUi{l,l);U2{l.l)\Uo{l)) 

E P[C] > Ec 2^ JoJiJ2'7(s)= ^ , rr r \ TT r- \ jt r- tTT r- ~- MI r- - wn r- sn X^, 7/0(1), 7/1(1,1), 7/2(1,1) 

^-^ V-^ - - 2H'*;^oUo),;7i(jo,ji),;72(jo,j2))+i(7/i(jo,/i);C/2(jo,j2)|C/o(jo)) '' ' ' 

(S^ JoJlJ22'(-^^°(^)-^^(^-^)-^-(^-^))+'(^^(^-^)^^-(^-^)l^»(^))x.,77o(l),7/i(l,l),7/2(l,l) 

- Z.'ZISJ ''""^'''''^Ec|7,„(i),7;,(ia),l/2(l,l) E]„J,J2 2<^;^o(Jo),t/i(Jo,Ji),t/2(io,i2))+.(t/i(Jo,ji);C/2(Jo,i2)|C/o(Jo)) 
^STnf^^W si I I 9«('s;C/o(l),77i(l,l),C/2(l,l))+i(7/i(l,l);7/2(l,l)|7/o(l))^ 

- 2^'?l*J^;7o:2(l,l,l)-'oJlJ2^ Xs,C/o(l),C/i (1,1), 7/2(1,1) 

s 

X (J0J1J2 + JiJ22*(^'^«(l» + ^^2<'■^Uo{l),U^il^)) ^ j^2'(.;7/o(l),C/2(l,l)) 

_^2H^;'^o(l),(7i(l,l),;72(l,l))+»(7/i(l,l);L/2(l,l)|7/o(l))^ (71) 

E/ u ) Xs,ua,ui,U2 (72.) 

q[S, 770, Wi, 772 j ^ ^ j-l2j(s;t,o) -|- (Jo Ji)-l2'('*'"o"i) + (JoJ2)^2*('*;"""2) + (JqJ^J2)-12»(*;"0«iU2)+»(«i:«2W 

l{do(5,.so([/o,C/i,;72)) > Do,di(5,si([/o,t/i)) > Di, d2 (5, 52(^70, t/2)) > D2} 

1 + J-l2HS;t/o) + M-^2<S.UoUi) + M-^2<S.U„U2) + J„(MiM2)^l2*(S;'^ot/i7/2)+»(7/i;772|(7o) 

where the main step dTOl l follows from Jensen inequality for the two-valued convex function f{x) = — on R+. 

X 

Equation (fTTT i follows from separating the cases This is discussed below. Lastly, equation ( f72] i follows from the 

facts that (C/o(l), t/i(l, 1), t/2(l, 1)) is generated according to quo1Ui\Uo1U2\Uo- 

Derivation of equation ( TtTI i; In the following we consider each of the three case mentioned above separately. 
But first observe that 



(73) 



-)l(s;«o,tii,tt2)+«("i;M2l"o) _ 



q(77o,77l, 772) 



g(77o)g(77i|77o)g(772|7lo) ' 

Case 1, jo ^ 1: 

JJ^C|7/o(l),C/i(l,l),7/2(l,l)^ 

= J2 '7K)'?(«i|wo)'Z(w2|«o)2*(^^"'""^'"^)+^("^^"^l"°) 
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- 1, (74) 

where we have used the fact that {Uo{jo),Ui{jQ, ji),U2{jQ, j2)), jo 7^ 1 is independent of 
(C/o(l), Ui{l, 1), C/2(l, 1)) and drawn from quo1Ui\Uo1U2\Uo- 
Case 2, jo = 1 and ji 7^ 1,^2 7^ 1: 

Ec|[/o(i),c/ui,i),t/.(i,i)2^(^^^"(i)'^^(i'^'^)'^^(i'^'^»+^(^^(i'^'^)^^^(i'^'^)l^''(i» 

q{U^){l),ui,U2\s) 



E 



'/(t^o(l)) 



^ 2«(^;'7o(i))^ (75) 

where we have used the fact that given Uo{\), (C/i(l, ji), f72(l, ^2)), ji 7^ l,j2 7^ 1 is independent of 
({7i(l,l),{/2(l,l)) and drawn from 9[/,|[/„(.|C/o(l)W,|[/o(.|t^o(l)). 

• Case 3, jo = ji = 1 and j2 7^ 1: Similarly we have, 

^ 2*(«;'^o(i),;7i(ia)) (7g-) 

• Case 4, jo = j2 = 1 and ji 7^ 1: Similarly we have, 

Ec|r/o(i),aUM).r/.(M)2'("^"^'^'^^^''^^^^'''^^'''»+'^^^^''^^^^^^^^'''^'^"^'» 

^ 2Hs;t^o(i),c/2(i,i)) (77) 

. Case 5, (jo, ji, j2) = (1, 1, 1): 

Ec|c/o(i),£/.(M).r/.(i,i)2'^^^^«(^''^^(^'^)'^=^^'^»+^<^^<^'^'^^=(^'^'l^°(^» 

^ 2»(s;C^o(i),;7i(i,i),;72(i,i))+»([/i(i,i);'72(i,i)lc/o(i))_ (78) 

Applying (|74l)-(|77| to (|70ll yields (|7T]>- ■ 

V. Joint source-channel coding 

To illustrate the application of our technique to joint source-channel coding problems, we study the problem of 
joint source-channel coding over MAC in this section. 

A. Joint source-channel coding over MAC 

Consider the problem of transmission of correlated sources {Si. S2) over a MAC depicted in Fig. |6] Let qy 1x1X2 
be channel transition probability. We prove a one-shot version of the result of Cover-El Gamal-Salehi iTTgl for this 
problem. 
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Fig. 6. Joint source-channel coding over MAC. 



Definition 9. An code for lossless transmission of correlated source over a MAC consists of (possibly stochastic) 
encoders (fk '■ Sk f-^ Xk,k — 1.2, and (possibly stochastic) decoder tp : y Si x S2- 

Theorem 9. Let K be the common part of Si and 82- Given any q(si, S2Tt, xi, X2) = 
q{si, S2)q{t)q{xi\si,t)q{x2\s2,t), there is a code for a single use of the channel whose probability of 
correct decoding is bounded from below by 

E (^1 + 2'^(Si\S2)-i(Y;Xi\X2,S2X) _|_ 2''('52l'Si)-»('i';X2|Xi,Si,T) _^ 2^(S^'S2\K)-i(Y;Xi,X2\K,T) 2'*(5'i.5'2)-»(i';Xi.X2)^ 

Moreover, loosening this bound gives the following upper bound on error probability of the code, 

P[i{Y-Xi\X2,S2.T)-h{Si\S2)<l , or 
i{Y ■ X2\Xi, Si,T) ~ h{S2\Si) < -1 , or 
i{Y-Xi,X2\K,T)-h{Si,S2\K)<-i , or 

i{Y-Xi,X2)-h{Si,S2)<l]+'^x2~\ (79) 

where 7 is any positive number. 

Remark 7. Second order analysis of the above bound can be obtained in a similar manner as the Gelfand-Pinsker 
rate region. We will include it in the next version of this draft. Its equations resemble the four constraints given by 
l\19]l . It subsumes the result of [6] whose region includes two additional constraints. 

Proof We only prove the lower bound on probability of correct decoding. Derivation of the loosened bound 
is similar to that of Gelfand-Pinsker and is thus omitted. 

Codebook generation: We employ a one-shot version of the codebook used in [19|. 

> For each k E IC, draw a symbol T{k) from pmf q{t), 

« For each si, draw a symbol Xi{si) from q(^xi\si,T{k{si))) , 

m For each S2, draw a symbol X2{s2) from q{x2\s2,T{k{s2))) . 

Encoding: Given sj, transmitter j sends Xj{sj). 
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Decoding: In contrast to Gelfand-Pinsker and broadcast channel, we use an SLC for decoding. Observing y, 
decoder uses the SLC PsiS2\y{si, S2\y) to find an estimate of (si,S2). The SLC can be formulated as follows: 

p £ IN qjSi, S2)qiy\Xi{si), X2{s2)) 

g(si,S2)2'''(^'-^i'*i''^2(S2)) 

Analysis: Observe that the joint distribution of random variables factors as, 

P{si,S2,y,Si,S2) = q{si,S2)q{y\Xi{si),X2(s2))PsiS2\Y{si,S2\y)- 

The probability of correct decoding can be bounded from below by P[C] > J2si S2 y ^(^Ij •^2, y, si — si, S2 — S2), 
hence we have: 



EP[C]=E q{s,,S2My\{X,is,),X2{s2)) '^J - ...(.^^.(.o^x^t..)) (^0) 

(si ,S2 )GSupp 
(si ,S2)eSupp 

where Supp is the support set of the pmf gs^Sg^ that is, Supp = {(si, §2) : gsiSal^ij ^2) 7^ 0}. By T{k) we mean 

T(fc(si)) = T{k{s2)), and the main step (|82] | follows from the Jensen inequality for the convex function f(x) = — 

X 

on M+. To evaluate ( |82] |. we first find an upper bound on the denominator. To do this, we first split the denominator 
into five terms as follows. 



Sl,S2 (»1 ,<i2)eSupp: 



(^1 ,S2)£5"PP: 



31 ,S2)eSupp: 
^"l^^l 

5] g(si,.-2)Ec|TW,Xi(.i),X2(.2)2^^^^^^(^^^'^^(^"=» 

S2?^S2 

+ q{si,S2)2<y'^^^'''>^^^^'^'>\ (83) 

We find upper bounds on each term of (|83] | separately. 

« Case 1: fc ^ k{— fc(si) = fc(s2)), where k ~ k{si) — k{s2), 

Ec|T(fc),Xi(.i),X2(a2)2^^^^^^^'^^'''^^'^»= E 9(i)9(a;i|5i,t)g(a;2|s2,t)2»(''^---^) (84) 



2^ g(t,a;i,X2|si,S2) (85) 

t,Xi,X2 '^^^^ 
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9(y) 

<iiy) 

where ( [84] | and dSST l follows from the fact that {T{k), Xi{si), X2{s2)) is independent of 
(T(fc),Xi(si),X2(s2)) and is drawn from gTgxi|SiT(-|si, •)'?X2|S2t(-|s2, •) = 9tXiX2|Si,S2(-|si. S2). 
Using (|87] | we obtain the following upper bound on the first term of (l83T l: 

q(si,S2)llic|T(fc),Xi(.i),X2(.2)2 ^ > ^ < 2^ 

(5l,S2)eSupp: (si,S2)eSupp ' 

k^k 

= 1. (88) 

Case 2: si ^ si, S2 7^ S2, but fc = fc(si) = k{s2) = k, 

Ec|T(fc),Xi(.,),^2(.2)2**^^''^(^^^''''^<^"^» - J2 qi^i\^i,T{k))q{x2\s2,T{k))2<y^-^--^^ (89) 

= 2^ q(xi,X2|si,S2,r(fc)) -^-^ (90) 



J2xi,x^ q{xi,X2,y\si,S2,T{k)) 
q{y\si,S2,T{k)) 



(91) 



(92) 



9(2/) 

where (l89T l and (|90] | follows from the fact that given T(fc), {Xi{si), X2{s2)) is independent of 
(Xi(si),X2(s2)) and is drawn from qx,\SiTi-\si,T{k))qx^\S2Ti-\s2:T{k)) = qxiX2\Si,S2,T{-\si,S2,T{k)). 
Using (|92] i we obtain the following upper bound on the second term of i 



E '/(^-i,^-2)Ec|T(.),x.(..),X2(.2)2^'-"^(^-^^^"^(^-^» < E <z(.-i,.-2|r(fc))^(^^^iliHl»(9^ 

(Sl,ii2)eSypp: ( ij , s 2 ) ESupp: ^'^^^ 

^ g(si,s2,;/|J'(fc)) 
qiy) 

(5l,ii2)eSupp: 
fc(si)=fc 

^ q{k,y\T{k)) ^^^^ 

liy) 

where (|93] l is due to the fact that T is independent of (5*1, 52 ) and ( |94b follows from the definition of common 
part, and q{k) = '=i.''2)esupp: g(si,S2)- 

fe(si) = fe 

Case 3: si ^ si, S2 — S2, hence k — k(s2) — k, 

Ec|T(fe).x,(.,).X2(.2)2''<^^''^^'^^^''^^^^»=E«(^i|^i'^(^))2'^'^'"'''^''" (96) 

Xi 

= E .(-iN-i, ^2, nk), X2{s2) fM^^^^^^ (97) 

Exi g(a;i,y|si,s2,r(fc),X2(s2)) 
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where ( l96l l and (l97l i follows from the fact that given (r(fc), X2(s2)), Xi(si) is independent of Xi{si) and is 
drawn from qx-,\SiT{-\si,T{k)) = qx,x2\Si,S2.T{-\si, S2,T{k), X2{s2))- Using ^ we obtain the following 
upper bound on the third term of (|83): 



^ q{si,S2) IEc|T(fe),Xi(si),X2(s2)2 



j(y;Xi(si),Js:2(s2)) 



< 2^ 9(s2)g(si|s2,r(fc)) — (100) 

Si :(si ,S2)GSupp 



,g(2/|si,S2,r(fc),X2(s2)) 

,q{si,y\s2,T{k),X2{s2)) 



< 2^ q[s2)q{si\s2,T{k),X2[s2)) (101) 

si:(si ,S2)GSupp 



<yg(52) '^ (102) 

= g(,^) g(;/l^2,r(fc) x,(,2)) ^ ^(^^^)2»(,,2,T(.),X2(.2))^ (103) 
q{y) 

where jlOOI l is due to the fact that T is independent of (5*1, S2) in the pmf q that we started with (this should 
be confused with the pmf induced by the code), and ( llOll l follows from the Markov chain Si — S2T — X2- 
m Case 4: §2 7^ S2, si — si, hence fc = fc(si) = fc, Using symmetry between Case 3 and Case 4, we have the 
following bound on the fourth term of (|83] |: 

^ g(si,s-2)Kc|TW,x,(s,),X2(.2)2*'^^''^^^^)'''^*'^" <9(si)2^^^^^^^'^('=^^^^(^^». (104) 

^i-C'Si >s2)eSupp 

In summary, we have the following upper bound on the denominator of (|82] >: 

Ec|Tw,x,(.,),x2(.2) E g(5-i,5-2)2^(^^^^(^"^)'^^(^"^» < i + qik)2<y-'™ + q{s2)2<y-'^^-w^is2)) 

Sl,S2 

+ g(s^)2*fe;«i,T(fc),Xi(.i)) ^ s2)2'(^'-^^("^)'-^'("'^^. (105) 



Substituting this in ( I82b gives: 

+ 5(si, S2|fc)"^2*(^''''^(''))"''(^'^i('*i)'^"(''"') 

+ g(si|s2)"^2*(2/;s2,T(fc),X2(s2))-»(a;Xi(si).X2(s2)) 

+9(s2|si)~^2'(^^'*i'^('')'^i(''i))"'(^^^i('*i)'-^2(''")) + 1)' 

_ ^2'lisi,S2)-l{y\Xi{si),X2{s2)) 

^ 2h{si,S2\k)-i{y;Xi{si),X2is2)\k,T{k)) 
_|_ 2Hsi\s2)-tiy;Xiisi)\s2,T{k),X2is2)) 
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_|_2'i(s2|si)-j(a;X2(s2)[si,T(fc),Xi(si)) ^ (106) 



Using this and the fact that {T{k), Xi{si), X2{s2)) is drawn from qTqxi\SiTi-\si, .)qx2\S2Ti-\s2, ■) = 
qTXiX2\SuS2i-\si,S2), we have: 



EP[C]> J2 E 9(^1, S2Mi)g(^iki,i)'?(a^2k2, 09(2/12:1, 2^2) (2''(^^'^^'-^(^'"^'"^) 

si,S2,y i,a;i,a;2 

_|_ 2'i(si:S2|fe)-«(j/;2:i,2;2|fe,t) _j_ 2''(''il''2)-j(j';2:i|s2, 4,2:2) _j_ 2'»(«2|si)-4(i/;X2|si,t,a;i) -f^ 1^ 
_ Iff 1^1 I 2/i(Si|S2)-i(l';Xi|X2,S2,T) , r)h{S2\Si)-i{Y:X2\Xi.,Si.T) 

— '^qSiS2TXiX2YiY2 ^ "1"^ 

_|_ 2HSuS2\K)^i{Y-XuX2\K.T) _|_ 2/1(51, S2)-»(y;Xi,X2)~j (jQy) 

This concludes the proof. ■ 
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